• Home
• All articles

Storage for the web

There are many different options for storing data in the browser. Which one is best for your needs?

Apr 27, 2020 • Updated May 7, 2020

Pete LePage

• Twitter
• GitHub
• Glitch
• Blog

Internet connections can be flakey or non-existent on the go, which is why offline support and reliable performance are common features in progressive web apps. Even in perfect wireless environments, judicious use of caching and other storage techniques can substantially improve the user experience. There are several ways to cache your static application resources (HTML, JavaScript, CSS, images, etc.), and data (user data, news articles, etc.). But which is the best solution? How much can you store? How do you prevent it from being evicted?

What should I use? #

Here's a general recommendation for storing resources:

• For the network resources necessary to load your app and file-based content, use the Cache Storage API (part of service workers).
• For other data, use IndexedDB (with a promises wrapper).

IndexedDB and the Cache Storage API are supported in every modern browser. They're both asynchronous, and will not block the main thread. They're accessible from the window object, web workers, and service workers, making it easy to use them anywhere in your code.

What about other storage mechanisms? #

There are several other storage mechanisms available in the browser, but they have limited use and may cause significant performance issues.

SessionStorage is tab specific, and scoped to the lifetime of the tab. It may be useful for storing small amounts of session specific information, for example an IndexedDB key. It should be used with caution because it is synchronous and will block the main thread. It is limited to about 5MB and can contain only strings. Because it is tab specific, it is not accessible from web workers or service workers.

LocalStorage should be avoided because it is synchronous and will block the main thread. It is limited to about 5MB and can contain only strings. LocalStorage is not accessible from web workers or service workers.

Cookies have their uses, but should not be used for storage. Cookies are sent with every HTTP request, so storing anything more than a small amount of data will significantly increase the size of every web request. They are synchronous, and are not accessible from web workers. Like LocalStorage and SessionStorage, cookies are limited to only strings.

The File System API and FileWriter API provide methods for reading and writing files to a sandboxed file system. While it is asynchronous, it is not recommended because it is only available in Chromium-based browsers.

The Native File System API was designed to make it easy for users to read and edit files on their local file system. The user must grant permission before a page can read or write to any local file, and permissions are not persisted across sessions.

WebSQL should not be used, and existing usage should be migrated to IndexedDB. Support has been removed from almost all major browsers. The W3C stopped maintaining the Web SQL spec in 2010, with no plans to further updates planned.

Application Cache should not be used, and existing usage should be migrated to service workers and the Cache API. It has been deprecated and support will be removed from browsers in the future.

How much can I store? #

In short, a lot, at least a couple of hundred megabytes, and potentially hundreds of gigabytes or more. Browser implementations vary, but the amount of storage available is usually based on the amount of storage available on the device.

• Chrome allows the browser to use up to 60% of total disk space. You can use the StorageManager API to determine the maximum quota available. Other Chromium-based browsers may allow the browser to use more storage.
• Internet Explorer 10 and later can store up to 250MB and will prompt the user when more than 10MB has been used.
• Firefox allows an origin to use up to 2GB. You can use the StorageManager API to determine how much space is still available.
• Safari (both desktop and mobile) appears to allow up to 1GB. When the limit is reached, Safari will prompt the user, increasing the limit in 200MB increments. I was unable to find any official documentation on this.

In the past, if a site exceeded a certain threshold of data stored, the browser would prompt the user to grant permission to use more data. For example, if the origin used more than 50MB, the browser would prompt the user to allow it to store up to 100MB, then ask again at 50MB increments.

Today, most modern browsers will not prompt the user, and will allow a site to use up to its allotted quota. The exception appears to be Safari, which prompts at 750MB, asking permission to store up to 1.1GB. If an origin attempts to use more than its allotted quota, further attempts to write data will fail.

How can I check how much storage is available? #

In many browsers, you can use the StorageManager API to determine the amount of storage available to the origin, and how much storage it's using. It reports the total number of bytes used by IndexedDB and the Cache API, and makes it possible to calculate the approximate remaining storage space available.

if (navigator.storage && navigator.storage.estimate) {
  const quota = await navigator.storage.estimate();
  // quota.usage -> Number of bytes used.
  // quota.quota -> Maximum number of bytes available.
  const percentageUsed = (quota.usage / quota.quota) * 100;
  console.log(`You've used ${percentageUsed}% of the available storage.`);
  const remaining = quota.quota - quota.usage;
  console.log(`You can write up to ${remaining} more bytes.`);
}

The StorageManager isn't implemented in all browsers yet, so you must feature detect it before using it. Even when it is available, you must still catch over-quota errors (see below). In some cases, it's possible for the available quota to exceed the actual amount of storage available.

Inspect #

During development, you can use your browser's DevTools to inspect the different storage types, and easily clear all stored data.

While working on this article, I wrote a simple tool to attempt to quickly use as much storage as possible. It's a quick and easy way to experiment with different storage mechanisms, and see what happens when you use all of your quota.

How to handle going over quota? #

What should you do when you go over quota? Most importantly, you should always catch and handle write errors, whether it's a QuotaExceededError or something else. Then, depending on your app design, decide how to handle it. For example delete content that hasn't been accessed in a long time, remove data based on size, or provide a way for users to choose what they want to delete.

Both IndexedDB and the Cache API both throw a DOMError named QuotaExceededError when you've exceeded the quota available.

IndexedDB #

If the origin has exceeded its quota, attempts to write to IndexedDB will fail. The transaction's onabort() handler will be called, passing an event. The event will include a DOMException in the error property. Checking the error name will return QuotaExceededError.

const transaction = idb.transaction(['entries'], 'readwrite');
transaction.onabort = function(event) {
  const error = event.target.error; // DOMException
  if (error.name == 'QuotaExceededError') {
    // Fallback code goes here
  }
};

Cache API #

If the origin has exceeded its quota, attempts to write to the Cache API will reject with a QuotaExceededError DOMException.

try {
  const cache = await caches.open('my-cache');
  await cache.add(new Request('/sample1.jpg'));
} catch (err) {
  if (error.name === 'QuotaExceededError') {
    // Fallback code goes here
  }
}

How does eviction work? #

Web storage is categorized into two buckets, "Best Effort" and "Persistent". Best effort means the storage can be cleared by the browser without interrupting the user, but is less durable for long-term or critical data. Persistent storage is not automatically cleared when storage is low. The user needs to manually clear this storage (via browser settings).

By default, a site's data (including IndexedDB, Cache API, etc) falls into the best effort category, which means unless a site has requested persistent storage, the browser may evict site data at its discretion, for example, when device storage is low.

The eviction policy for best effort is:

• Chromium-based browsers will begin to evict data when the browser runs out of space, clearing all site data from the least recently used origin first, then the next, until the browser is no longer over the limit.
• Internet Explorer 10+ will not evict data, but will prevent the origin from writing any more.
• Firefox will begin to evict data when the available disk space is filled up, clearing all site data from the least recently used origin first, then the next, until the browser is no longer over the limit.
• Safari previously did not evict data, but recently implemented a new seven-day cap on all writable storage (see below).

Starting in iOS and iPadOS 13.4 and Safari 13.1 on macOS, there is a seven-day cap on all script writable storage, including IndexedDB, service worker registration, and the Cache API. This means Safari will evict all content from the cache after seven days of Safari use if the user does not interact with the site. This eviction policy does not apply to installed PWAs that have been added to the home screen. See Full Third-Party Cookie Blocking and More on the WebKit blog for complete details.

Bonus: Why use a wrapper for IndexedDB #

IndexedDB is a low level API that requires significant setup before use, which can be particularly painful for storing simple data. Unlike most modern promise-based APIs, it is event based. Promise wrappers like idb for IndexedDB hide some of the powerful features but more importantly, hide the complex machinery (e.g. transactions, schema versioning) that comes with the IndexedDB library.

Conclusion #

Gone are the days of limited storage and prompting the user to store more and more data. Sites can store effectively all of the resources and data they need to run. Using the StorageManager API you can determine how much is available to you, and how much you've used. And with persistent storage, unless the user removes it, you can protect it from eviction.

Additional resources #

• IndexedDB Best Practices
• Chrome Web Storage and Quota Concepts

Thanks #

Special thanks to Jarryd Goodman, Phil Walton, Eiji Kitamura, Daniel Murphy, Darwin Huang, Josh Bell, Marijn Kruisselbrink, and Victor Costan for reviewing this article. Thanks to Eiji Kitamura, Addy Osmani, and Marc Cohen who wrote the original articles that this is based on. Eiji wrote a helpful tool called Browser Storage Abuser that was useful in validating current behavior. It allows you to store as much data as possible and see the storage limits on your browser. Thanks to Francois Beaufort who did the digging into Safari to figure out its storage limits.

The hero image is by Guillaume Bolduc on Unsplash.